Refactor loading entries and include baseloader #126
Conversation
DunklesArchipel
changed the title
There was a problem hiding this comment.
Pull request overview
This pull request refactors the entry loading infrastructure by introducing new resource creation functions and incorporating the BaseLoader for handling CSV files with complex structures. The changes improve the API design by separating concerns between resource creation and package building, and enable support for CSV files with multiple header lines.
Changes:
- Refactored CSV loading into separate functions for different resource types (tabular vs. pandas dataframe resources)
- Extracted field updating logic into a standalone
update_fieldsfunction - Modified
Entry.from_dfto work with in-memory pandas resources instead of temporary files - Added support for CSV files with custom encodings, delimiters, and multiple header lines
- Updated pixi version from v0.50.2 to v0.63.2 across CI workflows
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| unitpackage/local.py | Refactored resource creation functions, added create_tabular_resource_from_csv, create_df_resource_from_csv, create_df_resource_from_df, renamed create_df_resource to create_df_resource_from_tabular_resource, extracted update_fields function, updated create_unitpackage signature |
| unitpackage/entry.py | Extended from_csv with encoding/delimiter parameters, refactored from_df to use pandas resources, updated mutable_resource property with format validation, modified save method to handle pandas-to-CSV conversion |
| unitpackage/loaders/baseloader.py | Added documentation example for column_header_lines parameter |
| pixi.lock | Updated pixi configuration, removed editable flag, added pypi-prerelease-mode option |
| examples/from_csv/from_csv_multiple_headers.csv | Added test data file with multiple header lines |
| .github/workflows/*.yml | Updated pixi version to v0.63.2 across all workflows |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if len(unspecified_fields) != 0: | ||
| logger.warning( | ||
| f"Additional information were not provided for fields {unspecified_fields}." | ||
| ) |
There was a problem hiding this comment.
The unspecified_fields list is initialized but never populated. This warning will never be triggered. If the intention is to track fields from the original schema that were not updated with information from the provided fields, the logic should populate this list in the else branch of the main loop (line 339).
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 participant
Checklist
doc/news/.